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Mr . May , 

Attached is a copy 
of the task force report 
on effectiveness indica- 
tors that forms the basis 
of the OP Positive Indica- 
tors Program. This is 
forwarded for your infor- 
mation in response to 
suggestion by a couple of 
the Office Dir§<F^<!)!ni"Lat 
the recent DDA Management 
Conferenc< 



Distribution : 

1 - Each DDA Office Dir, 
with attachment 
except D/OP. 

1 - D/OP wo/att . 
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MEMORANDUM FOR: DD/Pers-P§C 
DD/Pers-SP 


FROM 

SUBJECT 

REFERENCE 


DD/Pers-R^P 
F. W. M. Janney 
Effectiveness Indicators 

Report of OP Task Force on Effectiveness Indicators 
June 1976 


1. As the new Fiscal Year is underway, this is to remind you that 
the system of effectiveness indicators is now operative. During final, 
discussions of the task force, it was decided to eliminate Indicator 
II-C, Alcohol Problems, which is now an CMS responsibility. It was also 
decided to modify Indicator IV- C to measure the number of employees 
taking optional insurance rather than the amount outstanding per employee. 

2. I shall expect to receive reports on the following schedule 1 


INDICATOR I 

- A. 


B. 

ii 

- A. 


B. 


c. 

’"lit 

- A. 


B, 

tv 

- A. 


B. 

C. 


V - A. 

B. 

C. 


Recruiting - Semiannually 
EEO Recruiting - Annual 

Reassignment - Annual 

Retirement Benefits Counseling - Annual 

Eliminate . 

Position Grade Control - Annual 
Review of Positions - Annual 

Employee Perceptions - Following survey 
Level of Effort Indicators - As appropriate 
Market Penetration (Insurance) - Annual 

Errors (Pers. Record-Keeping) - Monthly 
Processing Time of Transactions - Monthly 
Timely Statistical Reports - Monthly 


VI - Internal Management - As PDP and APP data become 

available 


3. Please send information 
Chief, Plans Staff, OP. 


copies of these reports, when made, 

STATINTL 


to 
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REPORT OF OP TASK FORCE CN 
EFFECTIVENESS INDICATORS 

June 1976 
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INTRODUCTION 

On instruction from the DBA, the Office of Personnel created a 
task force in the fall of 1975 to "devise systems and procedures to 
develop and establish a series of positive indicators against which 
program missions (functions) can be judged." 

The task force was comprised of representatives from each of the 
six program areas in the Office of Personnel. 1/ As a first step they 
were asked to assess their programs , identify their clients , and list 
potential indicators that would affect their clients* views of how 
well they were being served. As the activities are quite diverse, 
this produced a rather lengthy list of potential indicators which would, 
however, be of value to program managers. From the viewpoint of the 
Office, it was desirable to develop a much shorter list which yet repre- 
sented the range of programs. So the next step was to develop the 
shorter list which is presented in this report. 



I. CONSIDERATIONS 


The Office of Personnel shares with other elements of Agency 
management' the responsibilities for staffing, career development, and 
supporting employee morale. In the personnel field the ultimate test 
of management is the effectiveness of Agency personnel in the execution 
of Agency programs. The Office of Personnel alone cannot assure this 
effectiveness but poor personnel programs could do much to preclude it. 
Given the inevitable pressure of resource constraints and the consequent 
desire of management to assign the higher priority to the operating 
programs, it is imperative that a staff function, such as personnel, 
evaluates the effectiveness of its programs so that the available resources 
are well -used and so that the resources required for essential personnel 
functions may be requested and defended. 
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With this need in mind, it was possible to specify some criteria 
for developing indicators : 

a. On a selective basis, represent a broad range of Office 
activities. 

b. Represent a significant portion (30-60%) of Office resources. 

c. Indicate not how much but how well in aspects that pertain to 
objectives (where they' exist) or client satisfaction. 

d. Identify indicators that are based on client feedback or that 
themselves would influence feedback. 

e. Identify indicators that pertain to significant resources 
(don’t bother with activities that require few resources) . 

f. Differentiate between factors controllable by OP and those 
controllable outside of OP. 

g. Provide D/Pers some indicators that pertain to the interaction 
between OP and other elements of management ( system indicators) . 

h. Reject indicators that involve great additional expense in 
assembling data unless management perceives they are worthy. 

* ■ 

i. Subject the indicators to revision after use if required. 

These criteria have been applied, although not rigidly, in developing 
the indicators presented in this report. 

A complicating factor for the Office of Personnel is its unusually 
broad clientele — employees, components, Directorates, and the Agency 
as a whole. Some personnel services -- such as position classification — 
may produce conflicting perceptions of value among the employee concerned, 
his Directorate, and the Agency as a whole. It is important, and not 
necessarily easy, to identify the relevant client. 

II. SYSTEM INDICATORS 

The personnel side of Agency management already has an unusual 
two part tool that provides significant indicators of Agency performance 
and effectiveness in such key areas as placement, separations, promotions, 
training, EEO, and executive development. The elements of this tool are 
the Annual Personnel Plan and the Personnel Development Plan/ The 
monitoring of performance against expectation provides significant manage- 
ment insights. The allocation of responsibilities for performance is 
usually apparent. 
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Additional system indicators might be applied.' Given the high 
cost of recruiting and the decentralized nature of hiring, the Director 
of Personnel may wish to monitor the ratio between the number of ' 
applicants in process (over the course of time) to the number being hired. ' 

A growth in the ratio might indicate matters requiring management 
attention. A decline in the ratio would be highly desirable and the 
cause should be identified and encouraged. Along with this indicator, 
it would be useful to set a unit- cost type of indicator, such as the 
average cost per EOD (determined by dividing total "Personnel Input" 
costs* by the number of EODs) . Such an indicator could be expressed 
in constant dollars to pezmit comparison of changes over time so that 
trends of increasing or decreasing cost might be identified. Since costs 
of recruiting different types of personnel (professionals, clericals, 
blacks . . .) tend to vary significantly, some disaggregation of the cost 
data might be desirable. 

Much of the cost of maintaining the personnel record system is 
influenced by data errors in originating offices. Hence, it is useful 
to monitor the number of errors found in personnel transactions as a 
ratio to the total number and to take corrective action with major offenders 
Over tims reduction of the proportionate number of errors would be a 
positive indicator of a successful policy of turnback and training. 

III. EFFECTIVENESS INDICATORS FOR OP 

M uch of the work of OP is susceptible to workload measurement and 
therefore to unit-cost measurement.. Such measures are basically 
quantitative. Client perception of the effectiveness of OP is affected 
as much by the quality of service provided as the quantity, so measures 
of effectiveness must reflect qualitative considerations as well. A 
fai lure to do so would create incentive to increase quantity at the 
expense of quality. 

In the following discussion indicators of effectiveness are identified 
for each of the program areas. A deliberate effort has been made to focus - 
on a small number of indicators for the Office overall. Our success in 
developing indicators has been uneven, as indicated by the appended 
comments. 


Program Area I. Personnel Input 
Indicator A. Recruiting 

Semiannually, by general occupational title, calculate the ratio of 
applicants placed into process against manpower requirements t as stated in 
the Advance Staffing Plan. 


*The total costs attributed to the resource area for "Personnel Input." 
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Background: As recruiting against requirements is a major 
responsibility of the Office of Personnel, the provision of an adequate 
•flow of qualified applicants is a key objective. The Advanced Staffing 
Plan has some inadequacies as an operational statement of recruiting 
needs , but application of this indicator would generate improvements in 
the statement of requirements. Past experience suggests that requirements 
normally can be filled if the proper number of applicants are put in 
process. On an annual basis, the current norm for profess ional/ technical 
openings is 2.8 applicants in process per opening and for clerical, two 
for one. If the statement of requirements is adequate, application of 
the above indicator would help to identify for management attention 
problem areas in the recruiting effort. 

Critique: Hie indicator may be affected by lags between the 
estabTishment of a requirement and recruiting against it. Thus the 
indicator may show no applicants in process even though the appropriate 
recruiting efforts have been started and ray stimulate overcompensation. 
This problem is the basis for using a semiannual indicator rather than 
quarterly reporting. 


A second problem is that of defining the occupational groups, which 
should be relatively comprehensive. 

A third problem is that applicants may be listed under occupational 
titles that differ from those of the jobs in which they are actually 
placed. 

A fourth problem is that a successful effort to reduce the ratio of 
applicants in process to EODs (a system effectiveness indicator) might 
cause this indicator to signal an inadequate number of applicants in 
process. 

Indicator B. EEO- Related Recruiting 


On an annual basis, measure the number of Black and Hispanic- American 
professional/ technical applicants placed into process against manpower - 
requirements stated in the Annual Personnel Plan. 

Background : This indicator would signal success or lack of success 
in EEO- related recruiting. The current norm is 2.8 applicants in process 
to 1 required. 


Critique : The indicator suffers from several of the weaknesses of 
the preceding indicator. 


Program Area II. Personnel Operations 

/ 

Indicator A. Reassignment 

This is a compound indicator consisting of four annual measures: 
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1. The number of employees who have contacted SPD . concerning 

reassignment. ' 

2. The number subsequently reassigned. '' 

3. The number of terminated employees (voluntary or involuntary) 
whose reassignment cases have been closed by SPD within one year. 

4. Nuriber of employees who have contacted SPD concerning 
reassignment and whose cases are still active. 

Background: The objective is to develop an indicator of the 
effectiveness of the internal market. Since many of the cases that 
contact SPD may fall into the problem category it is to be expected that 
there will be many failures in the form of employees who eventually 
' terminate. Successful reassignments alone would not be a satisfactory 
indicator, for successful counseling may induce some employees to remain 
on the job. 

Critique: The indicator is not clean in the sense that CP has 
control. Obviously successful reassignment would involve factors 
external to OP. 

The choice of a one-year cutoff for previous contact with SPD con- 
cerning reassignment is arbitrary. Experience may indicate that many 
terminations occur between 12 and 24 months after contact. 

* As a compound indicator, it may be difficult to interpret for 
management purposes. 

Indicator B. Retirement Benefits Counseling 

Relate (1) the number of valid complaints concerning misinformation 
or delays in the provision of benefits information to (2) the number of 
retirees. ■. 

Background : The objective is to provide a measure of the effective- 
ness of retirement counseling. As valid complaints provide an indicator 
of reduced effectiveness, the objective is to reduce the number of 
complaints. (As some complaints may be without basis, it is necessary to 
sort out those complaints that can be substantiated from those that can 
not.) - ' 


Critique : The indicator will be very sensitive to the determination 
of whether a particular complaint is valid or not. 

It is implicit that there is a close relationship between the number 
of errors or delays and the number of complaints. One can imagine 
circumstances when complaints would be more likely to be registered. 
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As there is a close correlation between the number of retirees 
and the number receiving counseling related to retirement, it is .valid 
to use the number of retirees in this indicator. 

> 1 * * 

Indicator C. Problem Counseling -- Alcohol Problems 

On an annual basis, compare the number of employees receiving OP 
counseling related to problems of alcohol with the number terminated who 
have received such counseling. 

Background: The objective is to provide an indicator of the^ 
effectiveness of problem-oriented counseling. If an employee with a 
problem who has received counseling is unable to hack it, it represents 
a failure in several respects but not exclusively the responsibility of 
the counseling program. 

Critique: There is a time lag problem so that it would be possible 
in a year in which problem counseling is very effective to have many 
terminations which really result from previous years in which counseling 
was not so effective. 

Successful outcome in problem counseling would relate to many 
factors that are beyond OP control. The same is true of failures. 


Program Area III. Manpower Management 
Indicator A. Position Grade Control 

On an annual basis compare the' Agency’s actual change in average 
position grade with the average grade projection contained in the Agency’s 
budget. 

Background: The objective is to control the Agency’s position 
average grade so that it remains within the projection contained in the 
budget — which presumably would reflect programmatic and substantive ^ 
requirements . 

Critique: The actual change in average position grade would be 
determined by many factors outside of OP control, yet if that is under- 
stood the measure has value. 


In the event that the realized change does not equate to the budget 
projection, it would be useful to disaggregate the data enough to identify 
the major causes and' the components contributing. 

Indicator B. Review of Positions 

, 

On an annual basis, compare the number of positions reviewed for 
proper grade classification against the total number of Agency positions. 
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Background : A lengthening time between reviews of components and 
of their positions significantly reduces the validity of position 
descriptions and classifications. It is necessary to monitor progress 
in reclassification to preclude this. 

* • * 

Critique : This is a quantitative measure, not a qualitative one, 
so it would be possible to improve the performance in the indicator by 
degrading quality of position review. 


Program Area IV. Benefits and Services 
Indicator A. Employee- Perceptions 

Employee surveys should include sections to elicit feedback concerning 
levels of benefits and services provided, priority interests for new or 
expanded benefits and services and the adequacy of the administration of 
such programs . 


Background : Management interests are served when employees are aware 
of benefits and services available to them, are assured that management 
will listen to their concerns for benefits and services commensurate with 
their contribution to the Agency, and are reasonably satisfied with the 
way in which programs are administered. 


Critique : The linkages between benefits and services and motivation 
are not clearly established once certain minimum requirements for economic 
security and the working environment have been met. Surveys should be 
careful not to establish an expectation that there is open season for new 
or* expanded services and benefits. 

Hie level of adequacy of the administration of benefits and services 
will be a function of the resources provided, which will be relatively 
limited. Hence, some negative feedback is to be expected. The important 
aspect would be a trend toward a perception of worsening administration 
(longer times to process, more errors in processing) which management 
migjit seek to avoid. " 

Indicator B . Level of Effort Indicators 

Strictly speaking, these are not effectiveness indicators but might 
provide early warning of trends that would cause adverse employee 
perceptions. Examples would be measures of the average time to process a 
claim, the number of errors reported, the number of employees briefed as to 
the availability of benefits and services, the number of publications 
distributed and the size of their distribution. 

Background : Such indicators already exist and provide njanagement 
with some idea of the adequacy of administration of benefits and services 
given the available level of resources. The additional need is for an 
indicator that might alert management if its judgment of adequacy is at 
odds with employees' perceptions. 
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Indicator C. Market Penetration 

Where employee enrollment for benefits is voluntary, it is possible 
to measure the extent to which employees take advantage of the opportunity 
Thus, one could measure the average amount of UBLIC insurance outstanding 
per employee. 


Background : Such an indicator would measure progress in informing 
and educating employees in the benefits available. 

Critique: Unlike the salesman on commission, the objective is not 
to sell but to inform.' The final decision for such voluntary benefits 
should always be the employee's. What would a high value of UBLIC 
insurance per employee really indicate? 

. Program Area V. Personnel Record Keeping 
Indicator A. Errors 


By month and by year, measure the ratio of the number of errors in 
personnel transactions received for processing to the total number of 
transactions received. 

Background : A high error rate on personnel transactions received 
from components introduces unnecessary costs in OP operations. Keeping 
statistics on the error rate makes it possible to concentrate training 
where it is most needed and most effective. 

Critique : Though some cost is involved in keeping the necessary 
records, this indicator already has proven of value. 


Indicator B. Processing Time of Transactions 

Record in and out dates of transactions received for processing and 
report on a quarterly basis the average number of days required. 

Background : Processing of certain personnel transactions cannot 
be completed until the appropriate personnel folder is available. 

Delays often result because folders are not in file or required update 
material is not in the folder. Such delays create distortions in those 
personnel statistics that are based on processing date rather than 
effective date. The indicator provides management a tool to see if the 
problem is worsening or being kept under control. 

Critique : Control of this indicator lies outside OP with those 
offices that have personnel folders out on loan. Improvement of the 
indicator would rest upon developing procedures to speed the / circulation 
of such folders. 
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Indicator C. Timely Statistical Reports 

Report on a monthly basis the average time between a request for 
data from SRB and the providing of such data. • • ’ 

Background: The effectiveness of a central personnel statistical 
service"! s related to its ability to provide timely information. Otherwise, 
customers may seek their own data banks or decisions may be made with 
.inaccurate or incomplete information. 

Critique: The delays in providing information may be driven by 
a reed for programming and batch-processing in OJCS. The indicator would 
■probably be very sensitive to the type of request received and would not 
be stable from month to month as the mix changes. Even so, the indicator 
should be of management value, and its improvement would help to forestall 
decentralization of the personnel data bank. 

Program Area VI. Internal Management / 

The degree of fulfillment of personnel goals for OP within the APP 
and PDP provide! an effectiveness indicator for internal management. The 
Office may wish to establish supplementary goals for career management 
and counseling. ~ ' 
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